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CLIMATE FORECAST VERIFICATION VIA MULTINOMIAL STOCHASTERS 
Rudolph W. Preisendorfer 


0. Introduction 

In this work we develop a general approach to the problem of forecast verifi- 
cation in physical climatology. This problem has already been the subject of 
numerous studies (cf. e.g., Brier and Allen, 1951; Namias, 1953; Panofsky and Brier, 
1958; and their references). We are encouraged to make another essay in this 
direction because these studies have only presented partial solutions of the problem 
by omitting essential stochastic elements; or if the latter were included, then the 
appropriate common geometric setting of the forecast and its realization (the pre- 
dictand) was not developed. Moreover, some early studies of the problem have confused 
its formulation by introducing elements of subjectivity and qualitative reasoning 
into what is a matter requiring objectivity and quantitative reasoning. 

In what follows we will take the point of view that both the forecast and its 
realization must be treated within the same quantitative framework: the forecast 
will be viewed as the numerical spectficatton of values that a geophysical field 
(e.g., temperature, pressure, precipitation or some combination thereof) wilZ take, 
at some specified times in the future over a specified set of spatially distributed 
points. The predictand field will be couched in precisely the same framework and 
so will be wholly commensurate with the forecast field. For example, if the pre- 
dictand field is atmospheric pressure at n points of the U.S. mainland and the 
values of pressure are classified into r categories at each point, then so too will 
the forecast field be presented in r-tile form at each of these n points. 


Moreover, in what follows we will solve the problem of finding a suitably 


general reference forecaster, i.e., a verifier against which the skill of all forecasters 
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can be gauged. We will do this by choosing the stochaster as a worthy competitor 

of the forecaster. That is, we choose a stochastic forecaster (a person or device) 
which is assigned precisely the forecast problem faced by its competitor, and proceeds 
in a purely random way to solve it: both forecaster and stochaster, each in his own 
characteristic way, must predict the future state of the same geophysical field over 
the same set of spatial points and same set of future times. There tis accordingly, 

tn prtinetple, a stochaster asstgnable to each forecaster whose efforts are to be 
vertfied. 

For us, then, a verification of a forecast consists of two parts, namely the 
application of: (i) a quantitative measure of the degree of match between a given 
predictand and the forecaster's prediction; and (ii) a probability measure of attain- 
ing the same degree of match between the given predictand and the assoctated 
stochaster's prediction. In every practical instance these two parts of the veri- 
fication are required to be readily converted into tabular or graphical form. In 
particular the forecaster's skill may be depicted as a point (in a suitably dimensioned 
euclidean point- or subset-space) to which has been assigned a level of statistical 
significance via the performance of the competing stochaster at that same point. 

Thus when two different forecaster's skills are to be compared, this must be done 

on the sane geometrte-probabilistie background, namely that of their common stochastic 
competitor. In this way we can also solve the problem of comparing the relative 
merits of a wide range of possible different forecasters* all attempting to predict 
the same geophysical field's configuration over the same space points and same set 

of future times. This, obviously, requires the appropriate cooperative preliminary 
arrangements by two or more forecasters to insure that their recorded efforts will 


fall into the common geometric-probabilistic verification framework. 


* These can range from the simplest, such as the persisters and advecters, to the 
most advanced of current prediction strategies. 


We will use our general approach to develop several of these frameworks so 


as to attain a hierarchy of ever-increasing stringency, appropriate parts of which 
may be adopted by each forecaster, who, as his mastery increases, can then apply 
ever more rigorous tests of his forecast skills. Moreover, it will be possible 
for him to compare his skills with those of other forecasters who attempt the same 
forecasts in a common framework, such as any of those given in the hierarchy below. 

The general principles utilized in the present approach are of sufficient 
breadth so as to allow their extension to virtually every problem setting the 
climate forecaster may encounter. However, in the interests of brevity we will in 
this study explore only a specific class of stochasters, namely the class of multi- 
nomial stochasters. This class is already so broad that it will cover many, if not 
all, of the cases encountered in usual practice. Yet we should mention that there 
exist settings which require classes of stochasters that are not multinomial per se. 
For example, giobal skill scores for analog forecasts require stochasters that are 
not multinomial, but rather an immediate generalization of these, i.e., the multi- 
vectorial stochasters. It will be noted, however, that the verification principles 
(i), (ii) enunciated above are still applicable as guides to attain the appropriate 
forms of the match and significance quantities, now in the analog setting. 

Having attained a general objective and quantitative overview of forecast 
verification, one can now go on to apply it in various ways to the practical aspects 
of economics and administrative problems contingent on sound forecasts and their 
verifications. These problems, of course, are beyond the immediate scope of this 
study and will be reserved for a future time. Yet we wish to make one important 
observation in this regard: if we possess an objective, quantitative verification 
system of adjustable stringency, such as that developed below, then it will always 


be possible to extract from it auxiliary quantitative, or even qualitative measures 


of forecast verification applicable to the specific needs of the less quantitative 


fields of economics and administration. In other words, we can more easily, in 
such matters as these, descend the ladder from objectivity to subjectivity and from 


quantity to quality rather than ascend it, and we now possess the basis for such 


descents. 
te Forecaster vs the Mean Stochaster 


The simplest form of competition between forecaster and stochaster uses the 
expected value of the stochaster's performance as a point of reference. This is 
exemplified in the popular form of the skill Sy given by the Heidke formula (Brier 
and Allen, 1951): 


¢ = ee (1.1) 


where u is the number of O-class errors (number of correct predictions) made by the 
forecaster in a set of n forecasts, and u is the expected number of O-class errors 


made by the stochaster. 


In viewing (1.1) within the framework of our verification principles (i) and 


(ii) defined in §0, the quantitative degree of match between predictand and 


prediction is u, but the probability measure associated with the stochaster is missing. 


The stochaster's mean u is, of course, a statistical point of reference that serves 

in (1.1) to tell whether the forecaster has positive or negative skill according as 
u>u or u<u . But what is missing is some number (a confidence level, e.g.) that tells 
how much better or worse, respectively, the forecaster's efforts are than blind 
chance. Thus (1.1) serves only to tell whether one is doing better or worse than 
chance, but not by how much, in a probabilistic sense. 


Consider, e.g., the lists of skill scores in Table 9. These may be associated 
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with the following hypothetical situation: A temperature field (say) is to be 
predicted over the U.S. mainland at 99 selected points (cf Fig. 1). Thus n=99 in 
(1.1). The predictions are to be made by stating that, at each point, the temperature 
will be either above normal (A), normal (N), or below normal (B), where 'normal' is 
some previously established climatological mean. For this purpose the range of 
temperatures occurring in the record at each station is divided or partitioned into 
three equal classes (or intervals): one that contains the normal temperatures, 

and an interval each that contains the above normal and below normal temperatures. 

In this way the data have been 'terciled' at each point. Subsequently, the pre- 
dictions are compared with the actual temperatures realized at each of the 99 points. 
Let u be the numberof correct predictions (e.g., if A is predicted and A occurs, 

the prediction is correct). On the average, by chance, one would expect to guess 
1/3 of the temperatures at the points, so that 99x(1/3)=33=u in (1.1). If, e.g., 
u=41, then the skill S99 would be +.121. If u=33, then S99 would be 0, while 28 
correct would have a skill of -.076. If two forecasters attempt to predict the 

same temperature distribution over the same 99 points, then the one with the higher 
Sgq value may be judged to be the more skillful for that particular event. In the 
long run, the skill scores of one forecaster may average higher than the other, and 
hence Sg would give some measure of tne relative skill of the forecasters. But 

what about their absolute skills? It may happen that one forecaster is in the 

long run uniformly better than the other in the S99 sense, but that neither is better 
at forecasting events than a thrown die attempting to do the same job! In what 
follows we shall explore the ideas inherent in this last observation, with the goal 
in mind of attaining one form of an absolute measure of skill? against which forecas- 


ters' efforts may be pitted. 


2. Forecaster vs the Binomial Stochaster 

One way to improve on the skill score formula in (1.1) is to attach to Si the 
missing statistical significance of the score. This is done by assigning to the 
forecaster's problem a competitive stochaster. For example, if the physical field 
has n points at which it is to be predicted, and the predictions consist in specifying 
one of r possible values at each point, then the associated stochaster takes the 
following form (in the preceding example, n=99, r=3): at each point the stochaster 
chooses randomiy one of the r possible values. Hence the probability of choosing 
any one of the r values is 1/r. At the next point he starts again and independently 
of his previous decision, the stochaster chooses randomly from the r possibilities 
at that point. He continues this way through all n points. Now imagine that the 
predictand is depicted as n appropriately distributed dots in the following abstract 


diagram of the prediction problem: 


classes 


trials 


The open circles are forecasts by the stochaster. Sometimes he has a hit (circled 
dot) and sometimes not. Since his trials of choice are independent of each other, 
the probability of u correct predictions is Qyr)", The remaining n-u predictions 


are incorrect and have probability qr of occurring. The probability of this 


lvu l\n-u 
~ (t= : 


particular set of u correct and n-u incorrect predictions is tel The 


total probability P tu) of u correct and n-u incorrect predictions, regardless of 


i 


which u dots are circled and which n-u are not, is given by 


a ! ] ]\n- 
air ory» gh 2 > (2.1) 


where n!/u!(n-u)! accounts for the number of distinct ways the stochaster can achieve u 


correct predictions in the set of n dots. *q (2.1) defines the performance of the dino’: 


He can have only two outcomes: correct prediction, or wrong prediction. 


This probability function supplies the missing information needed in the use 


of (1.1) to gauge how much better are the forecaster's efforts than the stochaster's. 


For example, Table 10 lists* values of Pgg(u) and its cumulative probability function 


Qgq(u) eo (2.2) 


ao.me 


for the case n=99 and r=3. By (1.1) we can find the values of skill S, now associated t 


with u and u for the case n=99. Thus comparing Tables 9, 10, we see that skill scores 


of (say) +.106 or greater are statistically significant at the 95% level. The 


column '1' in Table 10 corresponds to the u column in Table 9. Another index of 


skill in Table 9 is the critical ratio (where o is the standard deviation): 


u-U 
oO 


Cqg(u) = 


which is closely related to the approximating gaussian distribution to (2.1) for 


large n. The skill number Sh or the critical ratio are evidently but two of an infinite 


number of equivalent apparent-skil] indicators. Moreover this skill S as reckoned 


by (1.1) changes with u and n, so that +.106 need no longer be associated with 


statistical significance at the 95% level. 


numbers ltke the ertttc 


The main observation to make here is that skfl? 


Cog (u) or like Sgo7S(u 5U,99) are by themselves not the true tndteators of fo 


ing skill. The true indicators (relative to the competing stochaster) are given 


* See Preface to Tables 10-15, just before them. 


ee 
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via the cumulative probabilities Qgg(u). Thus, associated with u=40 is Qgg(u)=.9433, 
(of Table 10) which says that 94.33% of the stochaster's predictions are below 41 
correct. Or putting it another way, for every 100 tries by the stochaster to attain 
41 or more correct predictions at 99 points, only 100 - 94.33 = 5.67 times (on the 
average) will he be able to do so. Hence if a forecaster consistently obtains 
u=41 or more as a score in the present experimental setting, he is doing well relative 
to the stochaster, i.e., blind chance. 

There is an important point illustrated here which is perhaps too implicitly 
buried in part (ii) of the verification principle of 80 and which we now draw out 
in detail: in practice the stochaster works very hard at establishing his level of 
performance; experiment after experiment (under fixed conditions) goes by as he 
gradually establishes empirically the P tu) distribution which we so glibly assembled, 
by logical argument, in (2.1). In an identical practical sense, a forecaster's true 
skill emerges only after a sufficient number of experiments have determined (under 
fixed conditions) his own P tu) distribution relative to that of the stochaster. 
If the forecaster is consistently skillful, his ‘scatter diagram' of predictions, 
when superimposed on that of the stochaster, will show some distinctive and favor- 
able form of departure from the latter. This will be illustrated in some discussions 


below. 


3. Forecaster vs the Trinomial Stochaster (unsigned errors ) 


The next step up the ladder of ever more potentially stringent verification 
tests brings us to the trinomial stochaster. Returning to the diagram in §2 we now 
look not only at the correct number of predictions by the forecaster and stochaster, 
but also the number of 1-class, 2-class, ..., (r-1)-class errors they may commit. 

A j-class error, 0<j<r-1, is committed if the prediction circle and predictand dot 


are in classes whose indexes differ by j. For example a O-class error (j=0) is a 
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correct prediction, a l-class error (j=1) is a miss by one class. Clearly, for an 
r-tile classification of the predictand values, there can be up to (r-1)-class errors. 
A trinomtal stochaster is a stochaster whose scores are registered in 
three categories, namely as O-class errors, l-ciass errors, and 2-class errors. The 
latter are all errors of class 2,3,..., up to r-1, lumped together. We thus see 
that the trinomial stochaste* is the next step higher than the binomial stochaster; 
the latter's scores are registered as 9-class errors and T-class errors, where the 
latter are all errors of class 1, 2, ..., r-1, lumped together. 
We can determine the probability aj that a stochaster may commit a j-class 


error, as follows. Clearly a= l/r. Another way to see this is to reckon ap as: 


y 
ea (prob. that stochaster chooses cell i)x(prob. that predictand is 
i=] 
in cel? i) 


Continuing in this way: 


r 
my ee (prob. that stochaster chooses cel) i) x (prob. that predictand 
j= 


} 
is in cell (i-1) or cell (i+1)) 


| ea Ext ot gs eg 1 ,l 
Sp Dre tpt te Get te Ge 
\cilpnceensashasen iene eee a 
(r-2) terms 
2(r-1) 
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§3 
Again, 


. 
a = f (prob. that stochaster chooses cell i)x(prob. that predictand is 
i 
in cell (i-2) or cell (i+2)) 


- tet het debe ete het he lh 


Near —y amimemniiitnierengl 


(r-4) terms 


2(r-2) 
r2 


From these we can guess the general pattern for the probability aj. namely: 


a, = 20D | yejert (3.1) 


This may be checked, and a formal proof devised, by considering in detail, e.g., the 


cases for r = 6, 7. Another check consists in seeing that the sum of the aj is unity 


As an example, if r=3, so that we tercile the field values at each point, then 
agl/3, a,=4/9, a5=2/9. 

Now suppose that, in the context of the diagram of &2, the stochaster makes 
N predictions. Let u, v, w be the resulting number, respectively, of O-class, 
l-class, and 2-class errors. The probability of committing each type of error singly 
at a time is, respectively ay» ay. and ay (=1-(agtay)). Hence the joint probability 


of u, v, w is 


n! 
P(u.vaw) = ulviw! 4% 


a) + a, + ag = 1 r-tile 
(classification) 


utvtwen 


It may be verified that we recover the binomial P tu) of (2.1) if we fix u and sum 


p(u,v,w) over all possible values of v,w. That is, we fix u; then 


(3.3) 


p(u,v, n-(utv)) 


This process of summation may be viewed in the diagram below which gives an overview 


of the trinomial stochaster's domain. By fixing u, we fix a plane through u and 


The summation in (3.3) is over the lattice points of 


parallel to the wi plane. 


line L. 


Ww axis 


(worst score) 


stochaster's average 
score (u,v,w). ther 
_7 scores spread out over 
the inclined u,v,w 
triangle 


w=0 —— 
u = intseay fixing the set of points summed over 
. n (3. 

perfect high skill region in u,v plane 

score) 


2h 


u axis 
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The scores of the trinomial stochaster are represented as triples (u,v,w) of integers 
u,v,w which sum to n. Hence the set of all possible scores lies on the finite tri- 
angular portion of the inclined plane through the three n-points on each axis. The 
probability of each score is given by (3.2). A perfect score is one for which u=n 
and v=w=0, i.e., the point on the u-axis, a distance n from the origin. The worst 
score is the n-point on the w-axis, and a score of intermediate skill is the n-point 
of the v-axis, The stochaster, after many experiments of length n, each experiment 
resulting in a triple (u,v,w), begins to accumulate a cloud of points on the inclined 
triangle and centered on the average point (U,v,w) = (nag. nay» naz). For example, 


if n=99, and we choose terciles (so that r=3), then (u,v,w) = (33, 44, 22). 


4, Forecaster vs the Trinomial Stochaster (signed errors ) 


Suppose we are not only interested in the number of j-class errors committed 
by a forecaster, but also whether his errors were above or below the predictand 
mark. That is, e.g., if the predictand in a tercile classification were 'N', and 
the forecast error were of class 1, we would like to know specifically if it were 
ejther A or B. We now design a stochaster that will score a forecaster on the 
basis of such 'signed' errors. In this manner we supplement the significance tests 
of §3 by giving a way whereby we can determine if the forecaster tends on the average 
to over-or-undershoot the mark. 

After going through the derivations of §3, the present derivation is relatively 
simple. (Refer to the first diagram of §2.) A positive [negative] j-class error 
l<j<r-1 is committed if the prediction circle lies j indexes above [below] the 
predictand dot. For example, if the circle is in class 3 while the dot is in class 1, 


there is a positive 2-class error. Patterning our reasoning on that in §3A, we can 


see that aj(+), the probability of a positive j-class error, is 


Ree eee 


§4 
a,(+) = (cod) O<jcr-1 (4.1) 
a me _ 
and similarly 
a jf} el) O<j<r-1 (4.2) 
j -2 - 
is the probability of a negative j-class error. For example, if r=3, then ap=1/3, 
a, (+)=a,(-)=2/9, a,(+)=a5(-)=1/9. 
Our test for predictive symmetry in forecasting is supplied by the trinomial 
stochaster whose elementary probabilities are 
a(0) = 1/r (4.3) 
r-1 1 
a(+) = 5 aj(+) = (1-2) (4.4) 
r-1 ] 
a(-) = 5 a,(-) = (1-2) (4.5) 
I* 


Here a(0) is the probability of a -ciass error. a(+) gives the probability of a 
positive-class error, while a(-) is the probability of a negative-class error. The 
joint probability p(u(0), v(+), v(-)) of u(0) O-class errors, v(+) positive-class 
errors and v(-) negative-class errors incurred in a set of n independent trials by 


the stochaster is 


p(u(0).vi+)v(-)) = crapreteyrerey a(t rac tay 


a(0) + a(+) + a(-) = 1 


(r-tile classification) (4.6) 


u(O) + v(+) + v(-) 
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It may be verified that we recover the form of P(u) of (2.1) if we fix u(0) and sum 
p(u(O), v(+), v(-)) over all possible values of v(+), v(-). That is, suppose we 


fix u(0); then 


n-u(0) 
p,(u(0)) % stale p(u(0),v(+), n-(u(0)+v(+))) (4.7) 


The process of summation may be visualized in the diagram below which gives an overview 


of the trinomial stochaster's domain. 


u(0) 


n (perfect score) 
axis of symmetry in the inclined plane 


u(O0) = constant, fixing the set of points 
summed over in (4.7) 


v(+)=0 v(-)=0 
_ stochaster's average score 


(u(0), v(+), v(-)) 


(worst positive 
score) 


— v(+) 


~ 


high skill 
region in 
v(+),v(-) 
plane (near 
origin) 


u(0)=0 


axis of symmetry 
in v(+),v(-) plane 
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We now have an axis of symmetry in the stochaster's domain, either on the tipped 
triangular area or in the v(+), v(-) plane, about which the stochaster's scores lie. 
For example, the expected (average) scores of the stochaster for the case n=99, 


r=3 are (na(0), na(+), na(-1)) = (33,33,33). 


§; Forecaster vs the Multinomial Stochaster; (the concept x?) 


We will now explicitly consider more than three j-class errors in our search 
for the significance of forecaster skills. Of course, we can no longer visualize 
the skills in simple geometric diagrams, but we gain instead a useful parameter, 
the x? value, belonging to the forecaster's performance. We shall turn this parameter 
back into our preceding analyses to help solve the problem of ordering the skills 
when given in trinomial form. Thus the following excursion into the domain of the 
multinomial stochaster, while of possible interest in later studies, is actually our 
present means of introducing, in a natural way, the x? quantity into the theory of 
the trinomial stochaster. 

We return to the first diagram of §2 and let the stochaster perform an experi- 


be respectively the 


ment of n independent prediction trials. Let Ups Uys sees Ung 


number of 0, 1, ..., r-] class errors he commits in that experiment. Let Ag» ays sees 
an.) by the elementary probabilities that he commits such errors, respectively. 


Values for these were derived in §3. Therefore we can in principle compute the joint 


probability for the r values usr 


(r - tile 
classification) 
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By approximating the factorials in this expression, using Sterling's formula, by 


writing 


(u; - na.) 
t. fign Setietgertelest' { (5.2) 
and by making some further algebraic reductions, we find that, to good approximation, 


de r-] 
1) exp{- xj} (5.3) 


P(UpsUys-+. U1) = (2nn) "2a aa a 
l= 


Y- 


In this way we condense all the j-class scores uy into a single number of the form 


xi x Se (5.4) 
r 


This quantity, as is well known,* is governed by the x°-distribution (using our r-tile 


notation): 


(x2) (0-3) /2 a 
Trey OF) dx?) = Gav aerraly d(x?) (5.5) 


Since the us are constrained to add up to n, there are only r-1 degrees of freedom 
associated with(5.5). 
For example, let n=99, r=3 and consider the signed errors of §3. Then 


ag=1/3, a,=4/9, a5=2/9, and we now have 


2 


2 2 
> . (u-33) , (v-4a) , (w-22 
x ee ee (5.6) 


ee 
ns To(x?)d(x?) = 4 eX d(y?) 


* See, e.g., Kenney, J. F., ‘Mathematics of Statistics' (part two), PD. Van Nostrand 
Co. N.Y. (1947) (7th printing). pl67 has a particularly clear derivation of the 


x* distribution's form from (5.3). 


| 
| 
© | 
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In this case we have two degrees of freedom. 
Equation (5.6) gives the probability of occurrence of all those triples 
(u,v,w) with x? values in the range (x?-'sd(x?), x?+sd(x?)). Since (as we shall see 
below) the set of all (u,v,w) having exactly some fixed x* value generates an ellipse 
in the uv plane of the diagram of §3, (5.6) gives the element of probability that 
the triples lie in an elliptical annulus defined by y*+#hd(y’). 
The approximation (5.6) must be examined for accuracy in our present work on 
the trinomial stochaster. This will be done in detail below (87). But for the 
moment, we can view (5.6) as a possible tool for ranking the skill of a forecaster. In 


general, for a specified n, ag» a,» ay, we can form the quantity 
u-U + v-V + wow (5 7 ) 
u Vv W 


where 


us Nap» V = Nay, W = Nay 


and compute the probability of the value x? associated with (u,v,w). One might 
expect that of two forecasts, the one with the greater x? value is the better, since 
its u,v,w values would depart on the average more from the mere chance values u,V,w 
than the other forecast. Unfortunately, this is not generally correct. Mere 
departure from the chance point (u,v,w) in the triangular score plane of §3 is not 
enough to insure high skill. As we have seen, triples near the point (n,0,0) are 

to be preferred by an ambitious forecaster. How to rank the skill value of points 


in the uv plane is an important and to some extent an elusive problem. It will be 


taken up next. 


a a i a a 


The Problem of Ranking Forecasting Skill in the Context of Trinomial Stochasters 


We shall, in the present context of trinomial stochasters, explore several ways, 
all more or less objective, in which we can make a judgment that a forecast is good 
or bad. 


A. x? Ellipses and their associated probabilites 


As we saw in §5, the x* value associated with a performance triple (u,v,w) 
resulting from a forecast can in turn have an ellipse and a probability associated 
with it. Without going through all the mathematics (given in §7, below) we can 
understand the connection between the ellipse and its probability, as follows. 

Let T be the set of all possible triples (u,v,w), O<u,v,w<n, enclosed by the 
scoring pyramid of §3. Since u,v,w can take on only integral values between 0 and n 
inclusive, there are, in all, exactly (n+1)(n+2)(n+3)/6 such triples in T. (For 
example, in the case of n=99, the number of triples is 171,700.) Fortunately, we 
need not work with all these triples in T, by virtue of the sum constraint (utv+w=n) 
on them. We may thus restrict our attention to a subset of them, say the u,v 
plane. This has only (n+1)(n+2)/2 points of interest (for example, in the case of 
n=99, the number of (u,v) pairs is 5050). Each of these points may be envisioned 
(cf the diagram in §3) as the projection of the triple (u,v,w), in the triangular 
plane, down onto its correspondent (u,v) in the uv plane. Some observations follow. 

1) To each projected point (u,v) in the uv plane we may uniquely assign the 
probability of its associated point (u,v,w), as given by (3.2). For example by 
Table A (with ay =1/3, a,=4/9, a=2/9 ,n=99) the point (33, 33, 33) has the probability 
.00017 and we assiqn this probability to (33, 33). The point (33, 44, 22) (the 
‘average’ point) has probability .90880, and we assign this to (33, 44). Thus every 
point (u,v) in the uv plane has a probability, namely that of the unique point 
(u,v, n-(utv)) above it on the triangular plane. 


2) To every point (u,v) in the uv plane there is assignable via (5.7) a 
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unique x? value, namely that of the unique point (u,v,n - (u+tv)) above it (For 
example, for n=99, and r=3, with agl/3, a,=4/9, an=2/9, the point (33,33) has 

x? = 8.2500, and the point (33,44) (the 'average' point) has x* = 0). The set of 
points (u,v) in the uv plane having a x? value not exceeding Xg form an approximately 
elliptical region about (u,v), the average point, as center and with a well defined 
total probability. (For example, with n=99, r=3, ag= 1/3, a,=4/9, ao=2/9, if we set 
x6 = 1.4621, it turns out that there are about 79 points within the ellipse associated 
with Xo (see Fig. 24) and moreover the sum of the probabilities of these 79 points, 
each probability reckoned via (3.2), comes to .50206.) Thus to each value of x’ 

we have assignable a probability, namely the sum of all probabilities of the points 
caught within the elliptical region defined by x’. 

3) Examples of the x*-ellipses may be seen in Figs 26, 27, 28, 29. In 
particular, in Fig 26 we show the six ellipses associated with probabilities .50, 
-80, .90, .95, .98, .99 for the case of n=99, 4=3, and ag l/3, a,=4/9, an=2/9. Thus, 
the outermost ellipse contains 99% of all the probability mass generated by the 
stochaster: that is, if the stochaster makes a large number, say 100 experiments 
at forecasting tercile values at 99 points with basic probabilities for 0, 1-, 
and 2-class errors given by 1/3, 4/9, 2/9, respectively, then on the average, 99 of 
his performance pairs (u,v) will fali within the ellipse. The ellipses in Fig 27 
may be described in the same way, but now for the case n=99, r=5 (i.e., quintiles) 


for which ag=1/5, a,=8/25, a5=12/25. 


B. Various performance regions in the trinomial domain 


We now may consider the problem of ranking skill scores, or of grouping them 
into regions of high or low skill. To fix ideas, consider Fig 2 which depicts the 
trinomial domain for the case of unsigned tercile errors (§3) in which n=99, a= l/3, 
a,=4/9, a5=2/9. u is measured along the horizontal axis, v along the vertical axis. 


The average point is (u,v,w) = (33,44,22). Point 0 is the projection (33,44) of 


this point on the uv plane. The line d-d therefore separates the total triangular 
region into two parts: those points (u,v) such that u>33 (have positive skill Sgq3 
cf (1.1)) and those points such that u<33 (have negative skill Sgq)- 

1) Suppose in Fig 2 we consider the region bounded by d-d, the heavy 
portion of the 95% ellipse, the u axis, and the diagonal line w=0. This is a roughly 
triangular region with a portion of an elliptical region removed. Any point (u,v) in 
this region has an associated x? for which its probability is not less than .95. 
Hence we may at first believe that points in such a region are statistically 
significant. Of course, this is a mater of definition. However, we may not wish 
to consider points on or near d-d in this region as indicative of great skill in 
forecasting. For while such points may occur very infrequently (about 24% of the 
time) a point such as (33,20) with only 33 correct predictions and 20 1-class errors 
(and hence 46 2-class errors) strikes one as indicative of rather mediocre skill. 
Nevertheless the region so defined is a candidate for high skill, and we can propose 
it for further study. 

2) Consider next the triangular region in Fig 2 bounded by the line c-c, 
the diagonal line (w=0) and the u-axis. Call this region 'A'. Recalling our 
discussion in §2, we know that a score (u,v) with u>4] occurs only 5% of the time 
during a stochaster's attempts to predict. That is, the set of all points (u,v) 
in the domain with u>41 has associated with it a total probability mass of .05. 
Notice, however, that there are points (u,v) along the dashed portion of c-c that 
fall rather deep within the 95% x’-ellipse. These particular points are clearly 
not significant on the 95% level relative to the partitioning of the plane by 
x°-ellipses.* This shows that using only u values to judge a skill (as in 52) may 


lead us to misjudge that skill. If we choose that subset of the total triangular 


* Observe that there are many subsets of the total trinomial domain whose points 
have a total probability mass of nearly .95. The complement of A and the 95% 
ellipse under discussion are but two such subsets. One determines the confidence 
level of a subset by simply totaling the probability mass within it using Table A. 


ORAS Pree ease ae 


domain consisting of the smaller triangular region A bounded by c-c, less the 


segment of the 95% y?-ellipse, we would then have a set of points (u,v) associated 


with relatively high skill. The statistical significance of the subset would be 


slightly larger than 95%. (The exact increment of the value, which is near 1%, is 


not of interest here. It would be found by adding up the probabilities of the 


points in the elliptical segment removed from the c-c triangular region A. This 


can be done with the help of Table A.) Thus we have another well-defined candidate 


for a high-skill region, this one a bit more stringent than in 1) above. 


3) The horizontal dashed line b is formed by cutting the pyramidal solid 


with a plane parallel to the uw plane at a value of v equal to 52, which is the 95% 


level for a binomial! distribution*with probability of success equal to 4/9, i.e., 


ay of §3. The inclined dashed line b is formed by cutting the pyramidal solid 


with a plane parallel to the uv plane at a value of w equal to 29, which is the 95% 


level for a binomial distribution‘ with probability of success equal to 2/9, i.e., 


a5 of §3. Together with the heavy portion of the 95% ellipse, these lines (even 


though they are generous in their restrictions) define a region of high skill some- 


what more stringent than the preceding region. Obviously, a still more stringent 


region is that defined by a-a, since it contains still less probability mass within 


its region. Similar regions are defined in Fig 3 for the case of r=5. 


Examples of performance by forecasters 
1) 


the scores of a forecaster denoted by 'A' in Table 1. 


Sprinkled throughout the domain of Fig 2 are fourteen points representing 


These scores are the results 


of actual forecasts of temperature over the 99 points of the U.S. mainland depicted 


For example, according to Table 1 the predictions of Winter '74 yielded 


th EIQ? TS 


u=42 correct predictions, v=37 l-class errors and w=20 2-class errors, and the 


associated (u,v) pair is denoted by '1l' in Fig 2. Observe that point 1? is not 


* See binomial probability Tables 11, 12. 


significant in any of the three senses 1).~3) defined above. Neither is point 7, 
associated with the summer of '75, significant. There are five points 2, 3, 9, 10, 
14 that have negative skill and which, moreover, are not significant relative to 
the 95% contour. The set of six points 4, 5, 6, 8, 11, 12, however, are 

outside the 95% contour and are situated in high skill regions. In particular the 
three points 8, 6, 12, especially the latter two, are outstanding forecasts. 

Point 12, the second most outstanding of them all, was the temperature forecast of 
the infamous winter of '77. These points are outstanding because they have 
relatively high u values (number of correct predictions). Moreover their 2-class 
errors are very small by virtue of being situated near the w=0 line. In general 

we May say that the higher the u value and the smaller the w value the better the 
skill, But there are exceptions, and we shall] explore this situation at the 
appropriate time later in this study (cf §6E). 

2) Another way of looking at Forecaster A's performance is shown in Fig 4. 

Here the same fourteen points are displayed in the signed-error domain, as defined 
in §4. v(-), v(+) are along the horizontal and vertical axes, respectively. The 
regions of various stringency are defined as explained in the diagram. Thus the 
area of least stringency is defined by the axes and the line d-d. Here we are 
asking the Forecaster to merely perform better than chance in obtaining the number 
of correct scores u which are measured along the axis normal to the diagram. 
Recalling the perspective view of the scoring plane in §4, it is clear that the 
closer in toward the origin a forecaster's score lies, the better is his effort. 
Notice that Forecaster A's two outstanding performances (points 6, 12) stand 
closest to the origin. The scattering of the fourteen points is generally well 
balanced: six are in the overshoot region (above axis x), eight are below, indicat- 
ing that Forecaster A's performance is generally not to over or under estimate in 
his forecasts. In this frame, eight points are considered significant and are 


circled. They all lie in the region bounded by the axes, c-c, minus the area in 
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the 95% ellipse. The region of highest stringency, that bounded by a-a, the 95% 
ellipse, and the axes, has only three points, 5, 6, 12. This situation should be 
compared with that of the three points 8, 6, 12 caught in the a-a region of Fig 2. 
This shows that measures of forecast significance, even in the present relatively 
objective setting, are near, but not quite, absolute. However, a way of ranking 
every pair of forecasts will be given below, and which can help remove this 
ambiguity (cf S6E). 

3) The diagrams in Figs 3, 5 are exactly analogous to those in Figs 2, 4, 
but now for the quintiled-data case. These diagrams have been included here to 
point up the remarks made earlier that the trinomial scheme of gauging the skil) 
of a forecaster can be made arbitrarily stringent. For example, according to 
Table 13, for the case of a binomial stochaster with probability of success 1/5, 
the 95% level of performance is 26 correct predictions out of 99. But suppose in 
such a quintiled setting we still demand 41 correct (as in the terciled setting of 
Table 10) to be the mark of a good forecaster. It is seen that u=41 in the context 
of Table 13 is virtually an impossibly high performance for a stochaster. However 
for an expert competing forecaster, u=41 in a quintile setting may not at all be 
impossibly high; it simply would set a relatively higher demand on that forecaster 
whose method has reached a state of development in which the terciled setting is 
not sufficiently stringent, not much of a challenge. This stringency manifests 
itself in Fig 3 by the closer proximity to the origin of the nested set of ellip- 
tical contours. Now, to get to the high u, low w places, the forecaster must 


exert himself considerably more to rise out of the bull's-eye of mediocrity. 


Dd. Examples of forecaster vs stochaster 
We shall now compare the relative performances of forecasters and stochasters 
in actual experiments at prediction of temperatures and precipitations over the U.S. 


mainland, 


— 
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1) Table 1 gives performances of Forecaster A in terms of unsigned scores 
(u,v,w) and also in terms of signed scores (u(0), v(+), v(-)), as defined in §83,4. 
For example, the prediction scores of Forecaster A for the winter of '74 are (42, 
37, 20) = (u,v,w) for the unsigned errors and (42, 13, 44) = (u(0), v(+), v(-)) for 
the signed errors. The pertinent connections between these errors are given below 
the table. Thus v(+) is the sum of the positive 1- and 2-class errors, while v is 
the sum of the l-class errors of positive and negative type. In a similar way we 
can interpret the remaining Tables 2, 3, 4 for forecasters B, C. D, respectively. 
All four forecasters were engaged in predicting the temperatures at the 99 points 
(of Fig 1) over the U.S. mainland for the fourteen seasons listed. These are 
summarized in Figs 7, 8, 9, 10. The results of their performances in predicting 
precipitation are summarized graphically in Figs 11, 12, 13, 14, and are tabulated 
in Tables 5, 6, 7, 8. 

2) These latter four figures (11, 12, 13, 14) are worth studying in detail. 
A first impression is that Forecasters A and B are considerably superior to 
Forecasters C and D in forecasting precipitation. Of the latter two it appears 
that D has more points of positive skill than C. Forecaster C has no points in 
any of the areas of high skill defined in PR B, C above. Similarly for D, who 
just barely has a significant point (no. 3) to show for his efforts. Forecasters 
A and B, however, each have seven significant points: 2, 3, 4, 5, 8, 9, 10 for A 
and 1, 3, 4, 5, 10, 11, 12 for B. It is remarkable that four of the points they 
share, namely 3, 4, 5 and 10 lie in just about the same places in each diagram. 
Also note that each has a common point, namely 7, nearly dead center on the bull's- 
eye, meaning, of course, a shared poor prediction (the winter of '76). This leads 
us to conjecture that Forecasters A, B and Forecasters C, D belong to two different 


classes of ability, and each one in each group is comparable in skill to the other, 


namely A and B are of comparable skill while C, D are of comparable skill. 
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3) Turning to Figures 7, 8, 9, 10, we compare the skills of the same four 
forecasters, now in their attempts to predict temperatures over the 99 U.S. mainland 
points and over the fourteen seasons listed in Tables 1, 2, 3, 4. Once again 
Forecasters A, B show definite superiority over C, D. Indeed, Forecasters A,B each 
have six significant points in high skill areas: Forecaster A has points 4, 5, 6, 8, 
11, 12 while B has points 3, 7, 8, 10, 11, 12. Forecaster A showed extraordinary 
skill at point 6 (spring '75), while B showed such skill? at point 3 (summer '74). 
Forecaster C has points 5, 10, 13 as significant above the 95% level using the x’ 
criterion. However, observe that these are all of negative skill, showing that a 
high x? value (such as may be encountered in a contingency table of classified 
observations and predictions) does not necessarily mean high skill. Forecaster D 
has point 10 above the 95% level, but its u value and v value are undistinguished. 

4) In Figures 15, 16, 17, 18, the same temperature s’ills in Tables, 1, 2, 
3, 4 for the four forecasters are plotted, now using signed errors (84). Thus the 
information in Fig 7 for A is viewed in a new way in Fig 15. The first impression 
is that Forecaster A tends to have balanced forecasts on the whole: the number of 
over estimates above the symmetry axis is six while those below are eight. Forecaster 
B has the same split but in the opposite sense. To help judge the quality of skill? 
of forecasters B, C, D, the reader may wish to lightly sketch in various regions 
of high skill, as defined in Fig 4, on the appropriate diagrams. The temperature 

skills viewed via signed errors in Figs 17, 18 are completely undistinguished. 
Forecaster C seems to have a scattering of eight significant points, but observe 
that they are not in the high skill areas. Once again, statistical significance 
in the x° value doesn't imply quality forecasts. 

5) The reader may by now have surmised that forecasters C, D are actually 
stochasters. We shalj describe how they made their predictions. 

Stochaster C had five dice before him. The faces that normally had numbers 


'l' and '2' were marked with 'B', the faces on each die normally marked with ‘3', 


'4', had 'N', and the faces normally marked with '5', '6' had 'A' in the case of 
temperature predictions. For precipitation 'A,N,B' were replaced by 'L,M,H'. To 
make a set of five predictions, the stochaster threw all five dice on a smooth flat 


table.* The symbols on the scattered dice were always read from left to right and 


recorded. Each such throw vurserefore produced five random predictions. The throws 
continued until an accumulation of 99 predictions had been made. Each of these 99 
predictions was then compared with its correspondent for the particular season under 
study whose actual temperatures or precipitations (in tercile form) had been compiled 
and listed beforehand for each of the 99 stations. From this point-by-point 
comparison, the class errors were calculated and tabulated. This process of throw- 
ing dice and comparing these results with each of the 99 observed field values was 
repeated until al] seasons had been gone through for each set of temperature and 
precipitation data. 

Stochaster D had before him an urn containing nine white balls. Three of the 
balls had the symbol '0' inscribed on them; two had '+1', two had '-1'; and one had 
'+2' while another had '-2' inscribed on it. The numbers of balls for each symbol 
are the relative frequencies with which the j-class signed errors, j = 0,+1,+2 occur 
for terciled data (cf §4). To make predictions the stochaster drew a ball at random 
from the urn. If it had ‘'0' on it, then it was recorded that he made a correct 
prediction; if it had '+?' or ‘'+2' on it, he committed +)-class or #2-class errors, 
respectively for that draw and it was so noted. In all, for a given season he 
made 99 independent draws from the urn. At the end of the 99 draws the number and 
type of signed j-class errors were totaled. From these, the unsigned errors could 
be found. for example for season 1, Table 4 shows he had the score (u,v,w) = (33, 
46, 20), obtained from the signed errors as indicated below the table. 

6) The differences in appearances between the scatter diagrams of C and D 


are readily explained: recall) that C had a more open pattern than D, signifying 


* There is no siqnificance to the number of dice used; they simply were available 
from a popular game of chance. 
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more scatter, greater variability from season to season. These differences are 
clearly due to their individual means of prediction. Stochaster C worked directly 
with the actual predictands, while D did something less direct and more abstract: 
each time D made a move he was driven by the odds of making an error; by contrast 
each time C made a move he was driven by the odds of making a speetfie predtetton 
(A, N, or B, e.g.). In other words, D had a theoretical deduction of a higher order 
built into him relative to C, so that, in a manner of speaking, each move by D was 
equivalent to several moves by C. Indeed, if we would let C continue indefinitely, 
his scatter patterns would relatively rapidly tighten like those of D and in the 
limit be described by the elliptical contours in the diagrams: 50% of his scores 


would eventually fall within the 56% contours, 80% within the 80% contours, and so on. 


E. Ranking performances by moments and y* 


We may supplement the x? value of a score in judging skill by the following 
considerations. 

1) In the diagram below, point 1 is given in the uv plane. If on the one 
hand we rise vertically from 1 so as to leave u fixed, we go to a point 2 which is 
clearly associated with greater skill since v increases while w decreases; in other 
words, we are decreasing the 2-class errors and trading them in for 1l-class errors, 
errors which are more palatable to the forecaster. On the other hand, if we move 


to the left of point 1, horizontally, so as to leave v fixed, we go to a point 3 


which is clearly associated with lesser skill since u decreases while w increases; 
in other words, we are decreasing the O-class errors and trading them in for 2-class 


errors. 


constant skill 


increasing skill 


am 


ecreasing skill 


There must then be an intermediate direction between that of segments 12 and 13, 

say 14, along which there is no change in the quality of skill. A moment's reflection 
would show that once we give numerical weights to the importances of the 1- and 2-class 
errors, this direction of constant skill is fixed. A natural assignment of weights 


may be made by defining the moment m of a trinomial score. We write 


'm' for Oru + lev + 2ew 


and cal] it the moment of (u,v,w) about u. We may envision the v and w scores as 


occurring on a lever 


thereby producing a turning moment; the values v, w act like masses and their distances 


1, 2 respectively act like moment arms. The object of a forecaster is to minimize 


§6 29 


this moment, to bring it down to 0, ideally. The smaller m is, the better his 


performance. Since u + v + w = n, we can write 


m= 2n- 2u-v (6.1) 


where n is the number of prediction locations, as usual. For a fixed n and m, (6.1) 


defines a straight line in the uv plane, namely 
v= -2u + (2n-m) (6.2) 


along which the moments of the scores are constant, and hence, by agreement, the 
points (u,v) have equal quality in the moment sense. 

2) As a result of this assignment of a moment to each (u,v) we can, with 
the help of the y* ellipses (introduced in ® A) lay down a coordinate net over the 
trinomial uv domain, By means of this network, shown, e.g., in Fig 6, we can locate 
points and assign to them relative ranks of performance. For example, on Figs 6, 6a 
we have placed the average* points (u,v) of the scores given in Tables 1-8. The 
average temperature scores are given on Fig 6, the average precipitation scores are 
on Fig 6a. It is seen that our earlier conclusions about the essentially equivalent 
skills of A and B and their superiority over C and D are graphically borne out using 
the present coordinate frame. The average points of A and B on Fig 6a lie essentially 
along the same moment line and on the same x? curve. Fach is clearly superior to 
C and D. However A and B find themselves between the 50% and 80% y° ellipses, as may 
be seen by comparing with Fig 6b, in which the 50% - 99% ellipses are also drawn in 
for comparison. Thus, on the average, the performances of A and B are mediocre. 
These average points are also drawn in as the circled crosses in Figs 7-14. In the 


latter set, the standard deviation of each average score is shown by means of a 


~ — “ 


* Thus u is the average of all u-points and v that of all v-points. 
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dashed circle with radius equal to the standard deviation and centered on the mean 
point. These statistics are summarized below Table 1. Observe that in these average 
temperature and precipitation tables, while A has a larger average u than B, our 
agreement to measure performances relative to x? and m in Fig 6 shows that their 
performances are essentially the same. If an edge has to be given to one over the 
other, for the present accumulation of scores, A may be judged slightly superior, by 
looking at the m scores below Table 1 or closely at Fig 6 and seeing that, while A 
and B lie on the same x? ellipse, A lies on a slightly lower moment line. At this 
stage of development of the prediction art, these differences are too small either 
to comfort or discourage A or B, respectively. Observe in particular that the average 
u score of A or B by itself places either forecaster quite close to the 95% level 
(cf Table 9). If, however, we look not only at the number of correct predictions, 
but also at the number of t-class errors {and hence implicitly the 2-class errors) a 
new perspective on their performances is attained: In general, a good average score 
should land in a high skill regton and with a relatively small standard deviation 
circle. Both forecasters therefore should be concerned with increasing their average 
u and v scores; it was these that placed them both in a rather undistinguished area of 
the skill diagram. Moreover, consistently predicting climate variations manifests 
itself in smaller (tighter) scatter diagrams. 

Thus we now have a reasonably objective framework in which to gauge forecasting 
skill as actual scores begin to accumulate and scatter diagrams begin to fill in. 


3) We may summarize the ranking procedure using m and x* as follows 


(u,v) = (u',v') if x? (u,v) = x°(u',v') 
m(u,v) = m(u',v') 
(u,v) > (u',v') if f m(u,v) < m(u',v') 
| regardless of x° 
| 


ior m(u,v) = m(u',v') 


_ and x*(u.v) > x? (u',v') 


In other words, two pairs (u,v), (u',v') are of equal rank if their moments m and 


x? values agree. Observe they need not be coincident to be of equal rank. Points 
C, D in Fig 6 are essentially equivalent. Also points 1, 2 are of equal rank. If 
the moments of two points agree, then we use x? to break the deadlock, the one with 
the lesser probability of occurrence (higher x*) being of higher rank; e.g., point 
2 is of greater rank than 3 in Fig 6. Therefore, in ranking points within a given 
region or set of points we give precedence to the moment of a score. This is clearly 
a convention (rather than a logical deduction) but one that is based on the intuitive 
interpretations of the scores u,v and their probabilities of occurrence. If a reader 
takes issue with this convention, then this means that he must (i) decide on a new 
relative weighting of v,w errors (and come up with an alternative to the moment m) 
and (11) decide on the relative importance of the new m, and y*. It may be that 
these relative weights and relative importances would vary with location over the 
uv plane. 

A word of advice can be made here, in conclusion: whatever one convenes as 
the method of ranking performances, fine differences and subtle nuances in scoring 
systems will be swept aside and be inessential in the face of truly superior or 
even just good forecasting. The present method of ranking appears tc go far beyond 
what has already been used. Perhaps then it is time to turn to the really basic 
problem at hand, the problem of forecasting, to devote more energy to improving that 
art, and perhaps gauging such efforts with the basically adequate ranking scheme we 


now have at hand. 


7. Construction of Tables A-E and EXP 

The graphical scoring charts we have used in our studies above are based on 
some simple analytical geometry and on probability calculations. These latter 
calculations are summarized in Tables A-E and EXP. They represent a fresh look at 


the x? quantity by calculating its values and their corresponding exact probabilities 


Ci, cdi 
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from the trinomial distribution for p(u,v,w) given in §3. In particular we compared 
the approximate probabilities of yx? as given by (5.5) with their exact counterparts 
given by (3.2) and saw that, except for certain noncritical regions in the uv plane, 
the classical cumulative probability distribution for yx? was adequate to serve as a 
base for our probability ellipses in the trinomial skill charts. We now discuss the 
construction of these tables for the benefit of those who may wish to explore 
analogous skill chart constructions for values of n and Ag» Ap» ay not specifically 


covered in this study. 


A, Table A 
One of the motivations of this calculation was simple curiosity: to see what 
the probability was for each of the 5050 possible triples (u,v,w) (ranging from 
(0, 0, 99) to (99, 0, 0) on the triangular scoring surface depicted in 83. Accord- 
ingly a computer was instructed to find p(u,v,w) via (3.2) to five significant figures 
for the tercile case: agel/3, a,=4/9, an=2/9, n=99, It turned out that many of the 
| triples with low u values (<14) and high u values (>54) had probabilities far below 
10°79, Removinj these from the computed list, we were left with 2644 triples whose 
probabilities or associated cumulative probabilities were 107 or greater. The 
range of these 2644 triples may be seen in graphic form in Fig 25, or directly in 
Table A, which begins with the triple (14,52,33) and ends with (54,35,10). The 
triples jin Table A are arranged in ‘alphabetical' order and may be visualized as 
progressing through the uv plane as shown in Fig 25. Along with (u,v,w) are given 
their y° values (in the column marked 'X2'), their probabilities (marked 'P(A)'), and 
their cumulative probabilities (marked 'CUM P(A)'). In order to understand the 


connections with later tables, we summarize the present calculations as follows, 


using the column headings: 
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TABLE A: 
(u,v,w) X2 P(A) CUM P(A) 
(ordered) (computed ) (computed) (computed 
from P(A)) 
MS: 
yields 


As we progress through Table A, we observe the x* values dipping in value, reaching 

a minimum, then rising again, over and over again. This may be explained graphically 
by looking at Fig 6 and imagining the paths taken through its domain as indicated 
schematically in Fig 25. As we start with (14,52,33) and move along the trajectory 
suggested in Fig 25, and at the same time keep an eye on the values of P(A), we see 
that CUM P(A) builds slowly, being fed invisibly by P(A) until, finally, at triple 
(23,38,38) the triples have probabilities larger than 10, and which go on to swell 


> again at (23,63,13). All 


to a maximum at (23,51,25) and then decrease down to 10° 
of this can be followed in imagination on Fig 25 by visualizing a probability haystack 
centered on (33,44) in the uv plane. Again and again the ordered triples (u,v,w) 

in Table A slice through the haystack, taking increasingly meatier chunks of prob- 
ability as the vertical traverses in Fig 25 get closer to the u=33 slice. As this 
slice is traversed (see p(15) of Table A) we finally attain the maximum value of 
p(u,v,w) in the entire table at the average point (33,44,22), namely p(33,44,22) = 
.00880. At this point, as the cumulative probability tally shows, we have accumu- 
lated half of the total probability mass. After this, the slices cut through the 
lower slopes of the probability haystack, decreasing steadily in content until 


eventually, as the traverse of slice u=54 is made, the final readable contributions 


to the total mass are made, 


Table B 

For this table we ordered the yx? values, encountered in Table A, in increasing 
order. As these y* values were ordered we simply carried along the associated triples 
(u,v,w) and P(A) values. The net result was a shuffled set of triples and probabil- 


ities. From the latter, as we went along, we added them up and formed CUM P(B): 


TABLE B: 


(u,v,w) X2 P(A) CUM P(B) 


(shuffled) (ordered) (shuffled) (computed from 
shuffled P(A)) 


ee eee 


yields 


The net result, CUM P(B), could be visualized as an ‘integration’ of P(A) using a 
polar coordinate frame with (33,44,22) as center. As we progressed from smaller to 
larger x? values we were sweeping up P(A) values in ever larger (essentially ellip- 
tical) regions about (33,44,22), and adding them together. Fig 24 shows the 50% 
ellipse enclosing about 79 points. These 79 points are represented by the first 79 
entries of Table B from (33,44,22) to (31,41,27) at which the total probability mass 
accumulated was .50206. The x? ‘radius' at this point is 1.4621. In this way we 
were able to associate to each x? its exact associated cumulative probability. This 
was the primary purpose of Table B. By the time we had moved out to x? = 76.0909, 
" 


we had essentially accumulated all probability mass (to within 19 ~), and could have 


truncated the table there. The region covered by the associated ellipse may be 


estimated from Figs 6 and 25. See in particular the points on Fig 25 for x? near 75, 


76. 


Table C 


This table is Table B now with ordered triples for easy look up of CUM P(B) 


at each (u,v,w): 


TABLE C 


| (u,v,w) P(A) CUM P(C) 
| (ordered) (shuffled 
CUM P(B)) 
(as in Table A) 


Table D 

To see how well the x?-ellipses (to be constructed below) embraced the 
accumulating probability mass as we swept radially outward from the center (average) 
point (33,44,2°), we returned to Table A and arranged P(A) in decreasing order. In 
this way we nibbled outward from the center of the haystack, accumulating probability 


in a natural way, going along the 'true' contours of the discrete haystack: 


TABLE D 


(u,v,w) X2 P(A) CUM P(C) 


(shuffled) (shuffled) (decreasing |(computed from 
order) decreasing 
order of P(A)) 


i 


yields 


- To see what we had, we immediately made from this: 


” 


Sl 
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E. Table E 
TABLE E 
(u,v,w) X2 P(A) CUM P(E) 
(ordered) (shuffled 


CUM P(D)) 
(as in Table A) 


A spot check was made at several points (u,v,w,) in the uv plane to see how 
well the cumulative probabilities agreed in Tables C and E. This would give a check 
on how well the x* contours could describe the enclosed probability mass. The 


reader is invited to do the same. To start him off, consider the following selection 


of points 
CUM P(E) CUM P(c) 
a) (33, 44, 22) .00880 .00880 
b) (38, 40, 21) .44133 .45115 
c) (40, 39, 20) -65669 .66549 
a) (44, 36, 19) .92697 . 93879 
e) (48, 33, 18) .99199 . 99438 


These points are shown on Fig 6 radiating outward from the origin. The agreement in 
cumulative probabilities is within one or two percent. Other checks along different 
lines show that we may use the y* value as a radial index in terms of which, within 
a few percent, we may characterize the probability mass within the x°=constant 
elliptical contour. This then supplied the rigorous basis for the nested elliptical 
contours in the skill score charts of this study. Any further constructions wishing 
to use smooth elliptical contours to summarize constant-y* regions of given pro- 
bability mass must satisfactorily pass this test. Otherwise the exact constant-\’ 


contours, which will likely be somewhat irregular, must be found by detailed plotting. 


tee as ee 
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F. Table EXP 
As a matter of simple curiosity we wanted to see how closely the x? distribution 


(5.5) approximated the exact trinomial probabilities yielded by (3.2). The form of 


EA Aas ni tal hE lat ti oh We atten fe 


(5.5) for the terciled trinomial case is obtained by setting r=3, resulting in 
= 2 
Tolx?)d(x?) = 4s eX d(x?) , (7.1) 
a simple exponential in the variable y*. How well does (7.1) describe the present 


state of affairs? In Fig 23 we show a plot of the exact values of p(u,v,w) for 


various x? values. For example, for y?=0 we have from Table B the probability of 


P(33,33,22) as .00880, and is shown on Fig 23. For y?=1-13, there are several 

triples associated with each value (cf. e.g., xy? = 1.0227). The range of probabilities 
associated with each x? is indicated by the vertical bar on Fig 23. This points up 

the important theoretical fact that To (x?) does not account for the multiple- 

* valuedness of the exact yx? relation defined by Table B. Moreover, a plot of the 
exponential in (7.1) in Fig 23 does not coincide with the visually-fit exponential 
going through the mass of points from Table B. 

To see how well the cwmuZative probabilities were given by (7.1), the computer 
was instructed to find 


n -A. 
(CUM EXP), = 4% exp [5+] AA,, n>1 (7.2) 
{= 


1 
sus = 
Agaxq = (X2), 


where n denotes the row of Table B. Here ne is the ordered yx? entry in row i, and 


LA, = A. - A;_y> with Ay70. The listing below compares CUM P(B) with (CUM EXP) as 


j 
found in (7.2), which simulates the discrete indefinite integral of (7.1). 
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CUM EXP 


07610 
10797 
15419 
23892 


40544 
52730 


63176 
78374 
86883 
92333 
95343 
97076 
98167 
98944 


99360 


This shows that the cumulative probabilities of y? in the third column, as given 


by (7.1)-(7.2) are reasonably good approximations to the exact values. Strictly speak- 


ing, as we saw in Fig 23, there is no one triple associated with a x* value, but 
actually several. Hence the exact displayed range of values of CUM P(B) for each x°. 
A similar comparison with CUM P(E) is possible, and shows the same degree of close 
agreement with CUM EXP. This indicates that for rough practical purposes we can use 
tables B, C, E, EXP interchangeably when assigning probabilities to x*. However, the 
exact table for this purpose is B or C. Table A is our basic table from which our 


numerical knowledge of p(u,v,w) springs. 
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1 8. Construction of the Skill Charts 

| ; The elliptical contours in the various figures in this study (as justified by the 
above results on Table C and Table E) may be found analytically as follows. Imagine 

the set of all points (u,v,w) in the scoring plane (cf diag. in §3) with a given 


fixed x? value. Thus we imagine all (u,v,w) in the plane such that 

u-u st vv + W-W ~ x? (8.1) 
: 

where U= Mag» V = May, W = Nay 


and Ap» ay» a5 are defined in 83. Since 


ta al lial iia cca a eth ce Cate 


utvtw =n (8.2) 


there is a corresponding set of points (u,v) in the uv plane having the same constant 


x? value. Using (8.2) in (8.1) and solving for v as a function of u, we find 


v= =z. » (u-u) + v + bo! { (1-ab) (ua)? * wb? } 5 
vtw (8.3) 


where a = 1+(w/u), b = 1+(w/v) 


The plus sign describes the upper half, the minus sign the lower half of an ellipse 


centered on the straight line defined by 


ve ee (wa) +9 (8.4) 
vtw 


| This is the straight line through the average point (u,v) and the point (n,0), the 


ae ss ae 
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point of maximum skill] in a trinomial diagram. The v intercept Vg occurs where u=0. 
In the case of n=99, ag=t/3, a,=4/9, an=2/9, Yo is given by: 
) 


= "0") 
V nbn peg ere Ba i822 OB. 


A general sketch of the ellipses in the trinomial setting is given below. It is seen 


that the ellipses are vertically sheared about the line given by (8.4). 


\ 
i 
| 


The horizontal limits u, of the ellipses in these diagrams are obtained by setting 


the term in curly brackets in (8.3) to zero and solving the resultant quadratic for u: 


WTS) (8.5) 


By construction, each vertical line (u=u,) js tangent to its ellipse where the line 
(8.4) pierces the ellipse. A study of Table A shows that the line (8.4) is the Tocus 


of maximum probabilities observed by making vertical] (const u) slices through the 


probability haystack based on the uv plane. 


pe 
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The formulations above serve also to define the ellipses in the signed error 
i 
i Ls diagrams, such as in Fig 4. We simply make the following assignations in (8.1) and 


related equations and carry out the resultant forms of the calculations: 


Unsigned errors Signed errors 
u, U pairs with v(-), v(-) 
Vv, Vv v(+), v(+) 
W, W u(0), u(0) 
ay a(-) 
ay a(+) 
ax a(0) 


Finally, to assign a probability to y* values for the purpose of labeling the 
ellipses with confidence level values, we used Table B as follows: we ran down the 
table until we encountered cumulative probabilities .50, .80, .90, .95, .98, and .99, 


and then simply picked off the corresponding x? values, which are summarized below. 


Cum prob. Assoc. x° From std. x? tables 
ed. f. 


50% 1.4621 
80 3.212) 
90 4.6667 
95 5.9394 
98 7.8030 
99 9.1667 


It can be seen that our exact \* values agree closely with those obtained from standard 


* 
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(but approximate) yx? tables for two degrees of freedom. But this agreement is not 
generally known a prfor’ for a given n and Ags ays a5. For this reason, the 
cautious chart designer would go through essentially the procedures described in §7, 
to find not only his own particular y’ values for (say) the above probabilities but 
also to see if the ellipses themselves are adequate to describe the regions in the 
uv plane with constant x° (recall the concluding remarks of S7E). 

Final checks on the accuracy of our computer graphics are made in Figs 24, 25. 
In particular, note how closely the annlytically detinad ellipses follow the dis-~ 


cretely determined points with constant y’. 
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SEASON 


Forecaster's TEMPERATURE Average Scores 


14.9 | 73,7 
16.4 | 77.4 
i 


TABLE 1 
TERCILED TEMPERATURE 
FORECASTER A 


— 


~ 
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ve 


= v, (+) + v,(-) 
we Vo(+) + Vo(-) 


Forecaster's PRECIPITATION Average Scores 


A | 39.3 144.0 | 11.2 | 75 

s 1 sn | 47.3 | 7.9 | 76.1 

88.7 c | 31.3 | 46.8 | 6.7 | 88.1 
» | 35.9 | 44.2 | 5.9 | 82.0 | 
ae pS AED BAR: Nek. 


TABLE 2 
TERCILED TEMPERATURE 


FORECASTER 8 
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: SEASON v(-) Vv Ww 
] Wnt 74 38 54 | 16 
2 Spr 74 52 40 | 21 
3 Sum 74 7 36 ] 
4 Fal 74 32 47 | 33 
5 Wnt 75 2 48 41 | 15 
6 Spr 75 24 0 64 | 2 
7 Sum 75 3 26 48 6 
8 Fal 75 5 25 50 6 ' 
9 Wnt 76 ] 51 58 8 

10 Sor 76 3 25 50 4 
2D) Sum 76 ts 14 38 | 14 
12 Wnt 77 ] 12 5] 4 
Spr 77 0 63 58 9 
Sum 77 8 63 42 | 43 
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TABLE 3 
TERCILED TEMPERATURE 


FORECASTER C (target: actual predictand) 


SEASON 


ant 74 
Spr 74 
Sum 74 
Fal 74 
bint 75 
Spr 75 
Sum 75 
Fal 75 
Wnt 76 
Spr 76 
Sum 76 
Wnt 77 
Spr 77 
Sum 77 
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v(+) = vy(+) + vo(+) 
v(-) = vy(-) + vol-) 
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TABLE 4 
TERCILED TEMPERATURE 
FORECASTER D (target: idealized predictand) 


TABLE 5 
TERCILED PRECIPITATION 
FORECASTER A 
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5 
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8 
9 
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i vy (-) + Vo(-) 


nos? Se Oona PRP HK OH PN 


~ 
oO 


oOo OnN TD ON fF WN — 


= vy (+) + vo(+) 


) = vy(-) + v9(-) 


TABLE 6 
TERCILED PRECIPITATION 


FORECASTER B 
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TABLE 7 
TERCILEO PRECIPITATION 
h 
| FORECASTER C (target: actual predictand) 
j pes cadet cadatcanaiiocs 4 Seen teen Esane. 
} | SEASON ; u(0) Vo (+) v(+) vy (-) Vole). | ¥(+) iw fw 
Rica ctepeacendlttee  asitt ee SA ACs Sarees ee ee 
| | Sum 74 28 13 46 19 6 25 52 | 19 
| 2] Fal 74 33 15 35 19 13 32 38 | 28 
| 3} Wnt 75 29 7 25 28 17 45 46 | 24 
| | Spr 75 35 7 25 26 13 39 44 | 20 
5 | 5} Sum 75 31 8 34 27 7 34 53 | 15 
‘| 6] Fal 75 34 9 32 24 9 33 47 | 18 
i 7 | Wnt 76 27 20 39 24 9 33] 43 | 29 
| 8| Spr 76 27 21 46 2) 5 26 46 | 26 
i | Sum 76 36 nN 40 16 7 23 45 | 18 
+ 10} Wnt 77 34 20 47 13 5 18 40 | 25 
114 Spr 77 35 36 20 8 28 51} 13 
12 |) Sum 77 26 28 37 8 35 571 16 
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TARLE 8 
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FORECASTER ND (taraet: idealized precictand) 


dy SEASON u(0) vy, (+) vo(t) | v(+) vy (-) a | 
a4 = 
. . snehimnenicatehinatatnibeeti ' Pe ) maticiehaiagloaatal peiaeasiietanditaieiitt sevnniinitial = 
i" 
| f Sam 74 34 3 7 30 22 45 | 20 
1 2] Fal 74 30 15 13 28 7 32 | 28 
“nt 75 4 2 3 20 ” 46 | 12 
i 4] Spr 7/5 34 a5 7 32 24 10 7] 
5} Sum 75 35 20 10 30 23 4c 
6] Fal 75 34 25 15 40 20 45 | 20 
. 7} wnt 76° fl 39 27 n 38 16 43} 17 
8] Spr 76 || 38 20 6 26 26 46 | 15 
i a] sum 76 40 D7 10 37 25 52 | 17 
/ 10} int 77 38 20 4 3A 2 41 2 
| 1] Spr 77 33 18 N 2a 23 11 | 25 
121 sun 77 |! 36 23 10 33 24 47] 16 
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TABLE 9 


SKILL SCORES S$ 
AND CRITICAL RATIOS C vs u 
CASE OF n=99, p=1/3, U=33, o=4.69 


u = No. Correct Skill Score Critical Ratio 
Predictions Se ee 1 
(O-class errors) Sg = (u-u) (n-u) Cog = ‘u-u)o” 
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Preface to Tables 10-15 


- * These tables are included for the reader's convenience. In 
particular, 'K' can stand for u, v, or w, as the case may be, 
when specialized to the notation of this study. Thus, we have, 


for terciled data: 


In Table 10 K corresponds to u, P9K) to Pgg(u)> 
CUM P(K) to Qgg(u)- 0.3333333333 = 1/3 

In Table 11 K corresponds to v, P(K) to Pgg(¥)> 
CUM P(K) to Qog(v), .0.4444444444 = 4/9 

In Table 12 K corresponds to w, P(K) to Pgg(w) 
CUM P(K) to Qgg(w), 0.2222222222 = 2/9 | 


Similarly, Tables 13, 14, 15 are for quintiled data, with K 


corresponding respectively to u, v, w, and 


0.2000000000 = 1/5 
0.3800000000 = 8/25 
0.4800000000 = 12/25 


Such tables are readily made up for other values of P and N. 
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PINOMIAL PROBAKILITIES 
P(K) = INS ZK E(NeK) PC ReAeR IE CLAP) e (Nen) 


N= 99 
Ps Ge3333333333 


K P(kK) CUM P(K) 1-CUM Pp (K) 
14 0.00001 0200001 0299940 
15 0.00902 0.00003 02099997 
16 0.00006 0.00010 0<e99990 
17 0.00016 0200025 0299975 
18 0.00035 0.00061 0099939 


ly 0.00075 0200136 0299864 
20 0.0015) 0eG0287 0.997123 
e 0.00284 0.006571 0299429 
22 0.00503 0601074 0098926 
23 0.00842 Ve01916 0.98084 


>| 
| 
| 
24 0.01333 0.03249 0296/5) | 
2 0.02000 U.0S249 0294/5} | 
26 0.02846 04208095 0691905 
27 0.03648 U.11943 0.88057 
28 0.04947 021€890 0683110 
29 0.06056 0.22945 Oe? 7055; 
30 0.07065 0.30010 0.69950 
31 0.0786? 0.37872 0.62129 
| 32 0.08354 0.4€2 76 0e53774 
33 0.08480 0254707 0245293 : 
34 0.0823} 0.62938 0.37062 
35 0.07643 GO. 7058) 02294)9 
36 0.06794 Ve7T7I375 VeLe6PS 
37 0.05784 0.83159 0.16841 
38 0.04719 0487877 0012123 
39 0.03690 0.91567 0208433 
| 40 0.02768 0.94335 0.05665 
41 0.0199] 0.9€326 0203674 | 
42 0.01375 0.97701 0202290 | 
43 VeV09L] G6 .98612 001396 | 
44 0.U0580 0695192 00080" 
45 1.00354 0.99547 0.00454 
} 46 0.00208 64295755 0.00245 
i at 0.00117 0.94872 0.00128 | 
43 0.000464 0.95936 0.00064 
99 0.00033 0.95969 0.0003) 
50 0.0001)7 Us9SYRS U.000)6 
5) 0.00008 04959993 0.00007 
S2 0.00904 9.95997 0.200003 
Sy U,0000°? 02985994 0.G00N% | 


SG 0.0000{ U.GS999 0-.b00n] 
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FROM CORY FURNISHED NODDQ age 


sad 


BINOMIAL PROBABILITIES 


P(K) = INI 7K! ( 


K P(K) 
23 0.00001 
P4 0.00007 
Ps 0.00004 
26 0.00008 
el 0.00018 
28 0.00037 
29 0.00073 
30 0.00136 
3) 0.00242 
32 0.00412 
33 0.00669 
34 0.01039 
35 0.01544 
36 0.02195 
37 0.02990 
38 0.03903 
39 0.04884 
40 0.0586) 
41 0.06747 


42 0.07454 
43 0.07905 
44 0.08048 


AS 0.07869 
46 0.07350 
a? 0.06667 
48 0.05778 
49 0.0481) 
50 0.03849 
51 0.02958 
Se 0.02185 
53 0.01550 
54 0.01056 
55 0.00691 
56 0.00435 
57 0.00262 
58 0.00152 
59 0.00084 
60 0.00045 
61 0.00023 
62 0.0001) 
63 0.00005 
64 0.00002 
465 0.00001 


HK) LV(PeeK IC CLP) He (N-K) ) 


N= 99 


P= 0244444464444 


CUM P(K) 
0.00091 
0.000902 
0.00906 
0.00014 
0.00032 
0.00069 
0.00142 
0.00278 
0.00521 
0.00933 
0.01602 
0.0264) 
0.04184 
0.0€379 
0.09370 
0.13273 
0-1E157 
0.24018 
9.30765 
0.38219 
0.46123 
0.54172 
0.62041 
0.49432 
« 76099 
0.81877 
0.86688 
0.90537 
9293446 
0.95680 
0297230 
0.98286 
0.989748 
0.95412 
0.99674 
0.9S826 
0.999)1) 
0.994956 
0.95979 
0.99990 
0298996 
OeSS99R 
0e9S999 


17CUM P(K) 
0-9999G 
0.99992 
0299994 
0-9998K 
0e99IbR 
02-9993) 
0-¥935R 
00997>> 
0299479 
0299067 
0.98398 
0.97359 
0295816 
029362} 
0290630 
0286727 
0281843 
075982 
0269235 
066178) 
0253877 
024582 
0237959 
0630568 
002390) 
0.18123 
0-133)2 
0209462 
0.06504 
0204329 
0.02770 
OeOlT 4 
02010277 
9e005Kn 
0-003>% 
Oo0DL74 
0-090xK9 
0600044 
0-000?) 
0-00010 
0200U04 
02090002 
0-0000} 
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1 FROM COPY FURNISHED TODDC 

: o TABLE 12 


PINOMIAL PROBABILITIES 


PUK) = ONEZK 2 (NK) DU CPHRK IC CL=P) #e (Nan) ) 


Sb sais ge epee 


K P(K) Cum P(K) 1-CUM P(K) 
6 0.00001 0.00001 0.99999 
7 0.00004 0.00005 0299995 
8 0.00012 0.00016 0299984 
9 0.00034 0.0005] 0.99944 
0 0.00088 0.00139 029986) 
1 0.00204 0.00344 0299656 
2 0.00428 0.00772 0e992PR 
RS. 0.00819 0.01591 0-98409 
14% 0.01437 0.02028 0.96972 
15 0.02327 0.05355 0294645 
16 0.0349) 0.08846 0e9¥1154 
17 0.04869 0e1232715 0086285 
18 0.06338 Oe20U0S2 0-7994R 
19 0.07719 0.27772 067222n 


20 0.08822 0436594 063406 
el 02909482 0.46076 0253924 
22 0.09606 0655682 0244312 
24 0.08313 0.72183 026817 


25 0.07125 0.80308 02196497 
26 0.05794 0.86102 013898 
28 = 0403288 = 069 3867 = 00 06193 
29 0.02300 0.9€167 0203832 


30 0.01534 0.97/00 0e0230n 
3] 0.00975 0.98676 0.01324 
32 0.00592 0.94268 0200732 
33 0.00343 0.9S6)1 0e-003K9 
34 0.00190 0.569802 0.00148 
35 0.0010) 9295903 0.000497 
36 0.0005) 0.99954 0.00046 
37 0.00025 0.9S979 0-000?) 
38 0.06012 0.9599] 0.00004 
39 0.00005 0.94996 0.00004 
40 0.00002 0e9SYGK Oe.VN0ND 


41 0.0009} 0695999 02060091 
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TABLE 13 FROM COPY FURNISHED TO DDG ome 


BINOMIAL PROEGABILITIES 
PUK) = UNE ZK EE (New) PL PeeK LE C1=P) ee (N-K) 


N= 99 
P= 062000000000 


P(K) Cum P(K) 1-CUM P(K) 
0.00002 0.00002 0299998 
0.00007 0.00009 0.9999) 
0.00023 0.00032 0299962 
0.00067 0.00093 0.99902 
0.00168 0.00267 0.99733 
10 0.00378 0.00645 0.9935¢ 
ll 0.007€E5 0.01410 0.98590 
12 0.01403 0.02813 0.97187 
13 0.02347 0.05160 0.94849 
14 0.03605 0.98765 0291235 
1S 0.05107 0.1387) 0.86129 
16 0.06702 0.20574 0679424 
17 0.08181 0e28755 0-7? 1245 
1g 0.09317 0.38072 0.61922 
19 0.09930 0.48002 0251998 
rs 0.09930 0.57932 0242068 
2l 0.09339 0.67271 0232729 
22 0.08278 0.75548 0624452 
ee: 0.06928 0.82476 0217524 
24 0.05485 0.87961 0212039 
25 0.04114 0.92075 0.07926 
26 0.02927 0.95002 0204998 
ras 0.01976 0.9€980 0.03020 
28 0.01272 C 98252 0.01748 
2 0.00778 0.299030 0200970 
31 0.00253 0.95737 0200263 
32 0.00134 0.9587) 0.00129 
33 0.00068 0.99939 0.0006) 
34 0.00033 0.99973 0.00027 
35 0.00015 0.995988 0.00032 
34 0.00007 0.9S995 0.00005 
37 0.00003 0.994999 0.00002 
38 0.00002 0.99999 0.0000) 
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TABLE 14 


BINOMIAL PROBABILITIES 


P(K) @ ONE 7KE (N=K) PLP eHK IL C1<P) (NK) ) 


K P(K) 
13 0.00001 
14 0.00003 
15 0.00007 
16 0.00017 
17 0.00039 
18 0.00084 
19 0.00169 
20 0.00319 
el 0.00564 
22 0.0094) 
2 0.01483 
24 0.02210 
2s 0.03120 
25 0.04179 
eT 0.05318 
28 0.06435 
29 0.07414 
30 0.08140 
31 0.08527 
32 0.08527 
33 0.08147 
34 0.07442 
35 0.96504 
36 0.0544) 
37 0.04360 
38 0.03348 
39 0.02464 
40 0.01739 
4) 0.01178 
42 0.00765 
43 0.00477 
44 0.00286 


45 0.00164 
47 0.00048 
48 0.00025 
49 0.00012 
50 0.00006 
51 0.00003 


Je 0.00001 


N= 9S 


Pz 023200000000 


Cum Pin) 
0.00001 
0.00004 
0.00010 
0.00028 
0.00067 
0.00151 
0.00321 
0.00639 
0.01204 
0.202145 
0.03628 
0.05839 
0.08959 
0.13139 
0618456 
022489) 
0.32305 
0240445 
0.48972 
0.57498 
0.65645 
0.73087 
0e7959) 
0.85032 
0.86392 
0.92740 
0.95203 
0.96943 
0.98120 
0.98886 
0295 363 
0.95649 
0.99814 
0.99905 
0.595953 
0.9SY977 
0.9S990 
0.95995 
0.96998 
0.95999 


1-CUM PI (K) 
0699999 
0.99996 
0.99990 
0299972 
0299933 
0299849 
0.99679 
0.9936) 
0098796 
0.97855 
0.96372 
0.9416} 
0.9104) 
0.3686} 
0261544 
0-75109 
06676956 
0.5955¢ 
0.510228 
0242507 
0234355 
02269} 3 
0220409 
021496A 
0-106€08 
0.07260 
0.04797 
0.03057 
0.01880 
0-O01114 
0-00637 
020035) 
0.001A4 
0.00095 
0-U0047 
0.00023 
02000)}0 
0-00005 
0.0000? 
020000) 
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PINOMIAL PROBABILITIES 
P(K) = ONE ZKE(NeK) EP OPO HK IE CL RP) He (NK) ) 


N= 9% 
P= 024&00000000 


K Pik) CUM P(K) 1=CUM P(K) 
el 0.0000) 0.00001 02999uN0 
23 0.00003 0.00004 0.999964 
ey 0.00007 0.000)) 0 -9998G 
30 0.00014 0.00025 0.99975 
31 0.00029 V.00054 § 0299944 
3? 0.000548 0.001)2 0-998R8R 
33 0.00108 0.00220 0299780 
34 0.00193 0.00413 0.99587 
35 0.0033) 0.00744 0.992564 
36 0.00543 0-01287 0-987)3 
37 0.00854 0.02142 0297853 
38 0.01286 0.03428 0.96570 
39 0.01857 0.05285 009475 
40 0.02571 0.07856 0.92lay 
41 0.03416 Vell272 0 e887>5 
42 0.04354 UelS626 0084374 
43 0.05328 0.20954 0079046 
44 0.06259 eclel3 Oel2e7ar 
45 0.07062 0634275 0265725 
46 0.07652 0.41927 0258073 
47 0.07965 0.498923 050107 
43 0.07965 0.257858 0042142 


49 0.07653 02655)0 00 344" 
56 U,V70E4 O.7é2574 0027426 
51 0.06265 ve 78839 Oe2llt) 
S2 0.05338 VeS4I177 0261582 


53 0.04370 0 e8E5S47 0211453 
54 0.03436 0.91483 02080)7 


SS 0.02595 Ve94578 020549) 
S6 0,0188° 0.96460 0203549 
7 0.01311 6.9777) 000274 
S58 0.00876 Ue4h647 0.0135 
59 0.00S5&° 0.96209 0.0074] 
60 0.09246 0.99555 0.00446 
61 0.00204 0.99759 0.0024) 
62 0.00115 UVeIS474 Oe Ol PE 
63 0.00063 0.99937 0.09063 
64 0.00033 0.959969 020003) 
65 ¥.00016 0695986 0.000)4 


66 0.06008 0.95993 0-00007 
67 0.00003 0.996997 0.00003 
68 0.00002 0.98998 0.00009 


69 0.00001 Ue9S999 00000) 
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